A Dynamic Memory Allocation Optimization Mechanism Based on Spark

Spark is a distributed data processing framework based on memory. Memory allocation is a focus question of Spark research. A good memory allocation scheme can effectively improve the efficiency of task execution and memory resource utilization of the Spark. Aiming at the memory allocation problem in the Spark2.x version, this paper optimizes the memory allocation strategy by analyzing the Spark memory model, the existing cache replacement algorithms and the memory allocation methods, which is on the basis of minimizing the storage area and allocating the execution area according to the demand. It mainly including two parts: cache replacement optimization and memory allocation optimization. Firstly, in the storage area, the cache replacement algorithm is optimized according to the characteristics of RDD Partition, which is combined with PCA dimension. In this section, the four features of RDD Partition are selected. When the RDD cache is replaced, only two most important features are selected by PCA dimension reduction method each time, thereby ensuring the generalization of the cache replacement strategy. Secondly, the memory allocation strategy of the execution area is optimized according to the memory requirement of Task and the memory space of storage area. In this paper, a series of experiments in Spark on Yarn mode are carried out to verify the effectiveness of the optimization algorithm and improve the cluster performance.     

基于NB分类器重访概率预测的Web缓存替换策略

Web缓存是用来解决网络访问延迟和网络拥塞问题，缓存替换策略直接影响缓存的命中率。为此，提出一种朴素贝叶斯（NB）分类器重访概率预测的Web缓存替换策略；根据用户之前访问日志，通过分区操作提取多项特征来表示每次访问的对象，并构建特征数据集；训练NB分类器，用来确定缓存中对象被再次访问的概率，为对象分配权重；结合LRU策略来合理删除一些对象。仿真结果表明，提出的策略在保证较高命中率的同时有效降低了执行时间。     

Optimised memory allocation for less false abortion and better performance in hardware transactional memory

ABSTRACT

This paper introduces and tackles a special performance hazard in Hardware Transactional Memory (HTM): false abortion. False abortion causes many unnecessary transaction abortions in HTM and can greatly impact the performance, making HTM not that useful when it is adopted as a fast path for Software Transactional Memory. By introducing a new memory allocator design, we are able to put objects that are likely to be accessed together from different threads into different cache lines and thus avoid conflicts of hardware transactions in different threads. Experiments show that our method can reduce 47% of transaction abortion and achieve a speedup of up to 1.67× (averagely 22%), yet only consume 14% more memory, showing great potential to enhance current HTM technology.     

基于日志统计特征的DNS隧道检测

以DNS服务器的日志为数据源,提取出二级域名的熵、子域名个数、缓存命中率等多维日志统计特征, 将日志量化为特征向量集;以特征向量集为数据源,使用随机森林算法进行模型训练,并使用十折交叉验证的方法对模型参数进行调整,对模型进行优化,提高整体检测精度;在不同分类算法下进行对比实验,并将实验结果与已有研究方法进行比较. 实验结果表明,提出的检测方法在召回率达到98.5%的情况下,有不低于90%的准确率,检测精度有所提高,即提出的算法能有效检测DNS隧道.     

Extending SSD Lifespan with Comprehensive Non-Volatile Memory-Based Write Buffers

Journal of Computer Science and Technology - New non-volatile memory (NVM) technologies are expected to replace main memory DRAM (dynamic random access memory) in the near future. NAND flash...     

Research on HTML5 application cache poison attack

HTML5 application cache (AppCache) allowed Web browser to access Web offline.But it also brought a new method of cache poisoning attack that was more persisting.As for websites which used the AppCache,a novel poisoning method RFTM (replace file twice method),in which the attacker replaced the manifest file twice to poison the client’s AppCache,was proposed.Compared with the original attack,the legal server would not receive abnormal HTTP requests from the client in the attack.Therefore,changing the server configuration could not prevent the client from the RFTM AppCache poisoning.To avoid the attack mentioned above,a lightweight signature defense scheme Sec-Cache in application layer was designed.Furthermore,experiments show that it has good performance and compatibility.     

异质存储系统中的高速缓存机制研究

存储子系统是嵌入式系统的重要组成部分.由于传统存储系统的设计已经无法满足容量日益增长的需求,固态存储器的应用越来越广泛.针对固态存储系统的存取速率慢的问题,目前常用的优化技术主要有缓存和并行读写技术.然而,在大多数应用和研究中,如何将这两类技术进行融合是目前遇到的一个重大挑战.对此,本文提出一种融合了缓存技术和并行读写...     

优化NoC性能的存储和通信压缩

文中针对NoC体系结构,提出了两种数据压缩技术,被称为高速缓存压缩和网络接口控制(NIC)内的压缩.性能实测结果指示压缩能够使NoC设计在较低的网络延迟、较低的功耗和改进应用性能等方面获得优势.     

高压缩V-LRU算法的研究与实现

提出并实现了基于四路组相联高速缓存的高压缩V-LRU算法。该算法将有效位和近似LRU标志位压缩到只有4位,可以大大减少电路面积,且高速缓存的缺失率基本保持不变。在高速缓存容量为8kByte时,高压缩V-LRU算法的缺失率与7-bit位比较近似V-LRU算法、5-bit位复用近似V-LRU算法基本相同,而相对于9-bit近似V-LRU算法也只增加大约0.9%。基于SMIC 0.13μm工艺,高压缩V-LRU算法的电路面积相对于9-bit、7-bit和5-bit V-LRU算法,分别减少10 925.8μm2、6 415.5μm2和2 142.1μm2。而且,如果增加高速缓存的容量,4种近似V-LRU算法缺失率的差别将变得更小,但是,高压缩V-LRU算法的电路面积优势将会更加明显。     

云计算分布式缓存技术及其在物联网中的应用

物联网是信息技术发展到一定阶段的产物,而云计算平台是物联网应用的基础。文章从当前云计算应用所面临的问题和缺陷出发,介绍了云计算分布式缓存的部署方式、功能架构及关键技术,并说明了分布式缓存高性能、高吞吐、高可靠性、高扩展性等优势和特性。文章解决了物联网应用普遍面临的数据可靠性、大容量内存共享、多模块数据一致保障、线性扩容等难题,为物联网平台云化架构的底层支撑奠定了基础。